13. Early Stopping

Early Stopping

As the above workspace shows, there are significant risks for peeking ahead and
making an early decision if it is not planned for in the design. If you haven't
accounted for the effects of peeking on your error rate, then it's best to
resist the temptation to look at the results early, and only perform a final
analysis at the end of the experiment. This is another reason why it's
important to design an experiment ahead of any data collection.

Note that there are ways of putting together a design to allow for making an
early decision on an experiment. In the workspace, we showed how to treat the
problem like a multiple comparisons problem, adjusting the individual test-wise
error rate to preserve an overall error rate. For continuous tracking,
this page describes a
rule of thumb for rate-based metrics, tracking the number of successes in each
group and stopping the experiment once the counts' sum or difference exceeds
some threshold. More generally, tests like the
sequential probability ratio test
can be developed to make an early stopping decision while an experiment is
running, if it looks statistically unlikely for a metric to move past or fall
back against the statistical significance bound.